CONTENTS Message from MMTC Chair

3D multimedia applications are receiving increasing attention from researchers in academia and in industry. This is in part due to new developments in display technologies, which are providing high quality immersive experiences for prices within the range of consumers. 3D video has become a strong candidate for the next upgrade to multimedia applications, following the introduction of High Definition (HD) video. This letter discusses approaches for error robust transmission of 3D video. It looks at some of the similarities and some of the differences between the approaches that can be applied to 2D and 3D video, and recommends some future directions for research. The focus is mainly on 3D video in the color plus depth format. Introduction The effects of transmission errors on the perceived quality of 3D video could not be less than for the equivalent 2D video applications, because the errors will influence several perceptual attributes (e.g. naturalness, presence, depth perception, eye-strain, viewing experience, etc), associated with 3D viewing. Therefore, 3D video content needs to be protected when transmitted over unreliable communication channels. Before considering how 3D video is made error robust, it is necessary to discuss the format of video that will be used. A plethora of formats have been developed to represent 3D video. The two representations currently featured in published standards are: • Color plus depth – where a depth map is sent alongside a color frame. The depth map represents the distance from the camera [1]; • Multi-view Video Coding (MVC) – a new standard developed by the Joint Video Team (JVT), which is capable of exploiting the correlation between the multiple views that are required to represent 3D video [2]. Work is being carried out combining the depth map with MVC [2]. A single view color plus depth video is sufficient to represent stereoscopic video, which is suitable for most of the 3D displays currently available. MVC can be used for holographic displays, and for Free-Viewpoint Video (FVV). In FVV, it is possible for the user to interactively select their viewpoint, and the view is then synthesized from the closest spatially located captured views [3]. This letter focuses on the color plus depth representation, due to its efficiency for the compression of 3D video, and its use in standards, such as MPEG-C. An important point to note is that the compressed quality of the depth map is not a significant factor in the final rendered quality of the stereoscopic 3D video [4]. This is because the depth map is not directly viewed, but is instead used to warp the 2D color image to two stereoscopic views. Furthermore, the human vision system obtains many depth cues from the structure of the 2D image. However, for FVV, the depth map quality becomes more important, as it is used to render virtual views that are further apart than with the stereoscopic case. When the views are further apart, then the distortion in the depth map has a greater effect on the final rendered quality. The rest of this letter considers the potential for 3D error robustness research. It splits up potential research areas into robust source coding, cross layer robustness, and error concealment. Robust Source Coding Many standard source coding approaches are available to provide robust source coding for 2D video, and many of these can be used for 3D color plus depth video. Features such as slice coding, redundant pictures, Flexible Macroblock Ordering (FMO), Intra refresh and Multiple Description Coding (MDC) [5] will all be useful. Perhaps the main outstanding issue is in deciding how to optimize their application. Loss aware rate-distortion optimization is often used for 2D video to optimize the application of robust source coding techniques. However, the models used have not been validated for use with 3D video. Although the importance of color compared to depth [4] would seem to indicate that existing 2D techniques will provide benefit, further work is needed to establish how the models can be better adapted. This is particularly true for FVV, where it would be beneficial to be able to model the effects of packet loss on synthesized free-viewpoints. IEEE COMSOC MMTC E-Letter http://www.comsoc.org/~mmc/ 12/28 Vol.4, No.3, April 2009 Cross Layer Error Robustness Considerable attention has been paid to cross layer optimization of 2D video quality in recent years. This has resulted in algorithms that have optimized channel coding, and that have prioritized the video data using techniques such as Unequal Error Protection (UEP), and prioritized resource allocation. Work has been carried out looking at a methods to transport 3D video across networks [6]. The study carried out in [7] proposes a JSCC scheme for color plus depth video. Prioritization of color over the depth information is also possible, as the color information has been shown to be more important than the depth map for subjective quality [4]. However, as the depth map can typically be compressed to 10-20% of the color information, the gains from this approach are limited. Error Concealment Most 2D error concealment algorithms can be exploited for 3D video. However, there is additional information that can be used in 3D video to enhance the concealed quality. For example, information, such as motion vectors can be shared between the color and depth video. If color information is lost, then depth motion vectors can be used to carry out concealment. However, there are many more opportunities with MVC, where adjacent views can be used to conceal a view that is lost. This can be through more simple disparity compensation techniques, or from more advanced warping using the 3D geometry of the scene. Summary This letter has described some of the issues associated with robust 3D video communications. Existing 2D error robustness techniques should offer improvements to the quality of 3D video transmitted over lossy networks. However, to obtain the best possible quality it is necessary to consider human perception issues (e.g. howperceived quality can be estimated), and theadditional data present in a 3D video bitstreamcompared to its 2D equivalent. References[1] W. H. A. Bruls, C. Varekamp, R. K.Gunnewiek, B. Barenbrug, and A. Bourge,"Enabling Introduction of Stereoscopic (3D)Video: Formats and Compression Standards," in Image Proc., 2007. ICIP 2007. IEEE Int. Conf.,Vol. 1 2007, p. I-89.[2] P. Merkle, A. Smolic, K. Muller, and T.Wiegand, "Multi-View Video Plus DepthRepresentation and Coding," in ImageProcessing, 2007. ICIP 2007. IEEE Int. Conf., Vol. 1 2007, p. I-201.[3] Y. Morvan, D. Farin, and P. De With,"System architecture for free-viewpoint videoand 3D-TV," Consumer Electronics, IEEETrans., vol. 54, no. 2, pp. 925-932, 2008.[4] C. T. E. R. Hewage, S. T. Worrall, S. Dogan,S. Villette, and A. M. Kondoz, "QualityEvaluation of Colour plus Depth Map BasedStereoscopic Video," Journal on Selected Topicsin Signal Processing: Quality Evaluation,Apr.2009.[5] H. A. Karim, C. Hewage, S. Worrall, and A.Kondoz, "Scalable multiple description video coding for stereoscopic 3D," ConsumerElectronics, IEEE Trans., vol. 54, no. 2, pp. 745-752, 2008.[6] G. B. Akar, A. M. Tekalp, C. Fehn, and M. R.Civanlar, "Transport Methods in 3DTV: ASurvey," Ccts. Sys. for Video Tech., IEEE Trans.,vol. 17, no. 11, pp. 1622-1630, 2007.[7] B. Kamolrat, W. A. C. Fernando, M. Mrak,and A. Kondoz, "Joint source and channelcoding for 3D video with depth image basedrendering," Consumer Electronics, IEEE Trans.,vol. 54, no. 2, pp. 887-894, 2008. Chaminda T.E.R. Hewage received the B.Sc.(Hons.) degree in Electrical and Information Engineering from the University of Ruhuna,Galle, Sri Lanka. During 2004-2005, he workedas a Telecommunication Engineer in the field of Data Communication. He was awarded his Ph.D.by the University of Surrey in 2009. His researchaims at providing QoS support for 3-D video IEEE COMSOC MMTC E-Letter http://www.comsoc.org/~mmc/13/28 Vol.4, No.3, April 2009communication application. He is a member ofIET and a member of IEEE. He was awarded agold medal by University of Ruhuna, Sri Lanka,for his achievements in Engineering discipline atthe general convocation held in 2004.S. Worrall received the MEng ElectronicEngineering degree from the University ofNottingham in 1998. He joined the Centre forCommunication Systems Research (CCSR), atthe University of Surrey, in 1998 as a researchstudent, and obtained his PhD in 2001. From2001-2003, he continued to work within CCSRas a research fellow, and was appointed as alecturer in multimedia communications in 2003.His research interests include error robust videocoding, multiple description coding, videocompression, transmission of video over wirelessnetworks, and multi-view video coding. He is a member of the IEEE and the IET. IEEE COMSOC MMTC E-Letter http://www.comsoc.org/~mmc/14/28 Vol.4, No.3, April 2009Editor’s Selected Paper Recommendation Ad hoc networks are infrastructurelesswireless networks with mobile users. These twocharacteristics make them excellent match forimportant military and civilian applications, allof which demand simplicity and flexibility indeployment and operations. However, the directconsequences of infrastructure independence and mobility are dynamic network topology andmulti-hop, fragile wireless paths, posing greatchallenges for provisioning of content-richmultimedia streaming services.A basic requirement of a streaming service is continuous delivery of media data, whichtranslates to continuous connectivity betweenmedia server and client. Furthermore, it would behighly desirable to have graceful degradation ofreceived media quality as network environmentchanges over time. The traditional approach ofaccessing a single server through a single pathcould hardly be adequate, since the server couldcrash or be unavailable due to high workload ornetwork partition, and the single path could bebroken or congested [1]. An effective solution tothese issues is service replication, as widely usedin the Internet to make service closer to clientsand for load balancing.Recent advances in Multiple Description(MD) coding have made it highly suitable forwireless multimedia communications [2]. MDcoding generates multiple equally important descriptions, each giving a low, but acceptablevideo quality. The decoding independenceamong the descriptions permits a reconstructionof video from any subset of received descriptions, achieving a quality commensurate with thenumber of received descriptions. MD coding ishighly suited for media streaming over ad hocnetworks, where links and servers are unstable,and reliable paths are hard to maintain [1,2].In this paper, the authors investigate theproblem of joint routing and server selection fordouble description (DD) video in ad hocnetworks. In addition to selecting a pair of servers, they also explore optimal routingstrategies to find high quality paths. Such a jointrouting and server selection scheme opens a newdimension of freedom for improving the DDvideo quality, since it explores a much larger solution space than existing server selectionschemes.Joint routing and server selection is firstformulated as a combinatorial optimizationproblem. This approach is application-centricand cross-layer in nature since the applicationlayer performance is optimized via network layeroperations. Due to the high complexity, exactsolutions are hard to find. Rather, the authors present schemes to compute a lower bound andan upper bound on the achievable videodistortion based on the monotone properties ofthe objective function. The upper boundproduces a near-optimal pair of servers and a pair of corresponding paths, while the lowerbound provides a benchmark for evaluating theoptimality of the upper bounding solution. Theproposed approach is computationally efficientand can be easily incorporated into existingrouting protocols for ad hoc networks.Extensive numerical results are provided toshow that the upper and lower bounds are veryclose to each other for all the cases studied,indicating their closeness to the global optimum.Significant gains in received video quality overexisting server selection schemes are alsoobserved, which justify the importance of jointlyconsidering routing and server selection for MDvideo streaming in wireless ad hoc networks.The idea of cross-layer optimized MD videostreaming has been extended in [3]. The authorsdevelop a branch-and-bound based framework,embedded with the Reformulation-LinearizationTechnique (RLT), which can produce (1-ε)-optimal solutions for any small value 0≤ε<1.